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ABSTRACT 

Adaptive  automation  (AA)  can  improve  performance  while 
addressing  the  problems  associated  with  a  fully  automated 
system.  The  best  way  to  invoke  AA  is  unclear,  but  two 
ways  include  critical  events  and  the  operator’s  state.  A 
hybrid  model  of  AA  invocation,  the  dynamic  model  of 
operator  overload  (DM00),  that  takes  into  account  critical 
events  and  the  operator’s  state  was  recently  shown  to 
improve  performance.  The  DM00  initiates  A  A  using 
critical  events  and  attention  allocation,  informed  by  eye 
movements.  We  compared  the  DM00  with  an  inaccurate 
automation  invocation  system  and  a  system  that  invoked 
A  A  based  only  on  critical  events.  Fewer  errors  were  made 
with  DM00  than  with  the  inaccurate  system.  In  the  critical 
event  condition,  where  automation  was  invoked  at  an 
earlier  point  in  time,  there  were  more  memory  and  planning 
errors,  while  for  the  DM00  condition,  which  invocated 
automation  at  a  later  point  in  time,  there  were  more 
perceptual  errors.  These  findings  provide  a  framework  for 
reducing  specific  types  of  errors  through  different 
automation  invocation. 
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INTRODUCTION 

Human  operators  are  increasingly  taking  on  the  role  of 
supervisor  in  complex  semi-autonomous  computer  systems. 
Yet  increased  automation  comes  with  unexpected  side 
effects.  This  is  because  automation  does  not  simply 
supplant  human  behavior,  but  rather,  it  interacts  with 
human  behavior  in  unintended  ways  [1].  This  can  include 
reduced  situation  awareness,  inappropriate  trust. 
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unbalanced  workload,  decision  biases,  and  over-reliance 
and  complacency  [2,  3,  4].  As  a  result,  there  is  a  need  to 
understand  the  types  of  errors  that  operators  make  when 
interacting  with  such  systems. 

Adaptive  automation  (AA)  has  been  proposed  as  a  solution 
to  the  negative  consequences  of  more-typical,  static 
automation  [5,  6,  7].  In  an  adaptive  automation  system, 
automation  is  flexible  and  responsive  to  the  needs  of  the 
user  and  the  changes  in  the  task  environment  [8].  AA  can 
be  invoked  by:  a)  critical  events;  b)  operator  performance 
measures;  c)  operator  physiological  assessment;  d)  operator 
modeling;  and/or  e)  hybrid  methods  [9].  While  empirical 
evaluations  have  shown  the  efficacy  of  adaptive  automation 
in  such  domains  as  aviation  [10],  air  traffic  management 
[11],  and  industrial  process  control  [12],  the  method  of 
invocation  can  dramatically  affect  how  the  operator 
performs  on  a  given  task  [13]. 

Of  these  methods  of  invocation,  recent  evidence  points  to 
the  potential  advantage  of  a  hybrid  invocation  method  to 
improve  operator  performance,  where  the  operator’s 
attention,  evaluated  by  analyzing  eye  movements,  and  a 
critical  event  are  used  to  invoke  automation.  This  was 
recently  demonstrated  by  the  dynamic  model  of  operator 
overload,  which  assessed  the  urgency  of  the  critical  event 
and  the  operator’s  attention  allocation  to  predict  whether  a 
vehicle  will  fly  into  a  hazard.  When  the  model  predicted  an 
error,  a  cue  was  initiated  on  the  situation  in  question,  which 
resulted  in  over  50%  fewer  errors  [14]. 

The  dynamic  model  of  operator  overload  (DM00)  was  an 
adaptation  of  the  fan-out  model,  initially  proposed  by 
Crandall  et  al.  (2005)  [15].  Fan-out  specifies  the  maximum 
number  of  vehicles  that  an  operator  can  effectively  control 
by  taking  into  account,  interaction  time  (IT)  —  how  long  it 
takes  to  interact  with  a  vehicle  in  order  for  it  to  be  in  an 
acceptable  state,  and  neglect  time  (NT)  —  how  long  the 
vehicle  can  be  ignored  before  it  needs  attention.  Variables 
that  captured  cognitive  limitations  of  the  operator  were  later 
included  in  the  fan-out  model,  such  as  wait  time  due  to 
human  decision  making  called  interaction  wait  time  (WTI), 
wait  time  due  to  attention  allocation  (WTAA),  and  wait 
time  due  to  attending  to  other  vehicles  that  were  ahead  in 
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the  queue  (WTQ)  [16]  (see  Equation  1). 

FO  =  (NT  /  (IT(\\T1)  +  UTAA  +  \\TQ))  + 1 

Equation  1:  The  Fan-out  model  initially  developed  by 
Crandall  et  al.  (2005)  and  later  modiHed  by  Cummings  et  al. 
(2008)  that  predicts  the  maximum  number  of  vehicles  that  can 
be  simultaneously  controlled.  WTI  is  wait  time  associated 
with  time  spent  on  decision-  making  and  is  nested  within  IT. 

The  DM00  uses  similar  theoretieal  eoneepts  as  the  fan-out 
model,  yet  instead  of  predieting  the  maximum  number  of 
vehieles  that  an  operator  ean  eontrol,  the  DM00  prediets 
operator  overload  in  real-time  during  the  performanee  of  a 
dynamie  task.  The  model  prediets  operator  overload  based 
on  whether  or  not  the  operator  will  respond  to  a  eritieal 
event,  sueh  as  whether  or  not  a  vehiele  will  interseet  with  a 
hazardous  area.  Three  theoretieal  eonstruets  from  the  fan¬ 
out  model  (see  Equation  1)  are  used  as  predietors  for 
operator  overload  in  the  DM00.  In  DM00,  WTAA  is 
instantiated  as  the  amount  of  time  it  took  to  look  at  objeets 
involved  in  the  situation  in  question,  WTQ  is  instantiated  as 
the  number  of  fixations  on  irrelevant  objeets,  and  NT  is 
instantiated  as  the  interval  of  time  from  when  the  vehiele ’s 
projeeted  path  first  beeomes  eritieal  for  the  operator  to 
address,  i.e.  when  the  vehiele  interseeted  a  hazardous  area, 
to  the  predieted  moment  that  the  vehiele  will  make  eontaet 
with  the  hazard. 

Another  eonsideration  when  developing  the  DM00,  and 
other  AA  systems,  is  what  eomponent  of  information 
proeessing  to  automate.  Parasuraman,  Sheridan,  and 
Wiekens  (2000)  [17]  proposed  four  levels  of  information 
proeessing:  sensory  proeessing,  pereeption/working 

memory,  deeision-making,  and  response  seleetion.  Eaeh  of 
these  four  levels  ean  be  automated  to  different  degrees 
where  at  the  lowest  end  of  the  speetrum  the  eomputer  offers 
no  assistanee  and  at  the  highest  end  of  the  speetrum  the 
eomputer  aets  autonomously  and  ignores  the  human.  For 
the  DM00,  deeision-making  was  automated  at  a  medium 
level  on  Parasuraman  et  al.’s  (2000)  [17]  seale,  where  the 
automation  suggests  an  alternative  aetion  by  eueing  the 
operator  to  a  situation  needing  attention.  The  reason  for 
automating  at  this  level  is  that  higher  levels  of  automation 
ean  have  negative  eonsequenees,  sueh  as  redueed  situation 
awareness  [11]  and  inereased  eomplaeeney  [18]. 

Due  to  the  unintended  eonsequenees  that  automation  often 
has  on  human  performanee  [17],  it  is  important  to  explore 
how  and  why  the  DM00  is  effeetive.  Sinee  inereased 
inaeeuraey  of  automation  typieally  results  in  worse 
performanee  [5,  19],  the  aeeuraey  of  the  DM00  should 
reduee  operator  errors.  However,  it  is  unelear  whether 
there  will  be  similar  types  of  errors  based  on  the  aeeuraey 
of  the  automation.  Three  types  of  errors  were  distinguished 
with  respeet  to  how  partieipants  respond  to  an  adaptive 
automation  eue:  (a)  The  partieipant  did  not  look  at  the 
situation  at  all  (Notieing  Errors),  (b)  The  partieipant  did  not 
look  at  the  situation  in  time  to  aet  (Time-out  Errors),  (e) 
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The  partieipant  looked  at  the  situation  in  time,  deeided  to 
aet  on  something  else,  and  did  not  aet  on  the  eued  situation 
in  time  (Delay  Errors). 

Additionally,  sinee  the  hybrid  method  invokes  automation 
based  on  both  a  eritieal  event  in  the  environment  and  the 
operator’s  state,  the  relative  eontributions  of  these  two 
faetors  in  redueing  operator  error  are  unelear.  In  partieular, 
the  timing  of  the  automation  is  a  major  differenee  between 
a  hybrid  invoeation  teehnique  and  a  eritieal  event 
invoeation  teehnique.  If  automation  is  invoked  based  solely 
on  eritieal  events,  the  system  does  not  have  to  take  into 
aeeount  the  operator’s  eognitive  state,  so  the  eue  ean  be 
invoked  mueh  earlier.  Thus,  when  eritieal  event  automation 
is  initiated  at  the  instanee  that  a  vehiele  needs  attention 
(Point  A  on  Figure  1),  the  operator  quiekly  beeomes  aware 
of  the  situation,  yet  may  deeide  to  aet  on  something  else, 
sinee  they  have  more  time.  But  with  the  DM00,  the 
automation  is  invoked  later,  resulting  in  an  urgent  seenario 
(Point  B  on  Figure  1),  whereas  the  partieipant  is  more  likely 
to  be  responsive  to  the  automation  and  aet  on  the  situation 
the  instant  it  enters  attention. 


Attention  is  Vehicle  Takes 

Urgent  Damage 

- L 

(B)  (C) 

Time 

Figure  1:  This  is  an  example  of  a  critical  event.  (A)  represents 
invoking  the  automation  based  solely  on  a  critical  event,  (B) 
represents  invoking  the  automation  using  the  DMOO  (B),  and 
(C)  is  the  negative  outcome  of  damage  that  occurs  when  the 
participant  does  not  respond  to  the  critical  event. 

Thus,  earlier  invoeation  of  automation,  sueh  as  eritieal 
event  invoeation,  may  result  in  partieipants  being  more 
likely  to  look  at  the  automation  but  deeide  to  aet  on 
something  else  first.  A  result  of  looking  at  other  objeets 
after  looking  at  the  eue  is  that  prospeetive  memory  errors 
and  planning  errors  are  more  likely.  Prospeetive  memory 
errors  involve  memory  for  intended  aetions  that  are  planned 
to  be  performed  at  some  designated  point  in  the  future  [20]. 
In  the  DMOO  eondition,  prospeetive  memory  and  planning 
errors  may  be  less  eommon  beeause  it  is  a  more  urgent  eue. 
Yet  sinee  the  DMOO  eue  is  generally  triggered  later,  there 
is  an  inereased  risk  of  errors  due  to  the  partieipant  not 
notieing  the  eue  in  time. 

We  therefore  hypothesize  that  the  timing  of  a  eue  will  affeet 
the  proportion  of  operator  error  types,  where  an  earlier  eue 
will  result  in  more  delay  errors,  and  a  later  eue  will  result  in 
more  notieing  and  time-out  errors.  Thus,  we  prediet  that 
the  distribution  of  error  types  will  not  be  different  when 
eomparing  an  aeeurate  DMOO  invoeation  system  with  an 
inaeeurate  DMOO  system,  where  both  systems  invoke  the 
automation  at  the  same  time.  Though  we  would  expeet 
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overall  performance  to  be  better  when  the  automation  is 
more  accurate,  based  on  previous  research  regarding  the 
negative  consequences  of  inaccurate  automation  [5,  19]. 
However,  because  the  timing  of  DM00  and  pure  critical 
event  invocation  will  likely  differ,  we  hypothesize  that 
there  will  be  differences  in  the  distribution  of  error  types 
between  these  conditions.  In  the  DM00  system,  where 
automation  is  invoked  later,  we  expect  that  there  will  be 
more  noticing  errors  and  time-out  errors.  Yet  for  the 
critical  event  invocation  system,  where  automation  is 
invoked  earlier,  we  expect  more  delay  errors.  The  reason 
for  this  is  that  when  the  DM00  invokes  the  automation  at  a 
later  point  in  time  there  is  an  increased  likelihood  that  the 
participant  will  not  notice  the  event.  However,  when  the 
critical  event  invocation  initiates  the  automation  at  an 
earlier  point  in  time,  this  increases  the  likelihood  that  the 
participant  will  look  at  the  situation,  decide  to  act  on 
something  else,  but  will  have  a  prospective  memory  or 
planning  failure  that  results  in  not  returning  to  the  critical 
event  in  time. 

EXPERIMENT  1 

To  determine  the  role  that  accuracy  of  invocation  plays  in 
the  distribution  of  errors  in  a  dynamic  task,  we  adopted 
Breslow  et  al.’s  (under  review)  DM00  and  ran  it  in  the 
same  simulation  they  used.  Similar  to  Breslow  et  al.  (under 
review)  the  DM00  predicted  whether  or  not  a  vehicle  will 
take  damage  by  providing  a  probability  that  damage  will 
occur.  When  the  probability  reached  a  high  enough 
threshold,  a  cue  was  initiated.  The  accuracy  of  the 
automated  cue  was  then  manipulated  by  changing  the 
location  of  the  cue.  In  the  accurate  condition,  the  DM00 
model  was  run  in  real-time  and  when  the  model  predicted 
damage  would  occur,  a  cue  flashed  on  the  relevant  hazard 
of  the  impending  damage  situation.  In  the  inaccurate 
condition,  the  same  DM00  model  was  run,  but  instead  of 
flashing  the  cue  on  the  relevant  hazard,  a  random  hazard 
(one  of  the  18  hazards)  flashed  when  the  model  predicted 
damage.  This  resulted  in  approximately  the  same  number 
and  timing  of  cue  invocations  in  both  conditions,  but  very 
different  levels  of  invocation  accuracy. 

METHOD 

Participants 

Twenty-nine  George  Mason  University  undergraduate 
students  participated  for  extra  credit.  All  participation  was 
voluntary.  Three  participants’  data  were  eliminated  due  to 
experimental  error  that  involved  an  issue  with  the  eye¬ 
tracking  hardware  not  recording  data.  Four  participants’ 
data  were  eliminated  due  to  low  eye  tracker  validity.  In 
total,  22  participants’  data  were  analyzed.  All  participants 
had  normal  or  corrected-to-normal  vision. 

Materials 

The  Research  Environment  for  Supervisory  Control  of 
Heterogeneous  Unmanned  Vehicles  (RESCHU)  [21]  was 
used  in  this  study.  A  Navy  pilot  who  is  familiar  with 
supervisory  control  tasks  designed  the  RESCHU  task  and 


the  task  has  been  used  extensively.  In  this  version  of  the 
RESCHU  task,  homogenous  unmanned  aerial  vehicles 
(UAVs)  moved  on  a  computer  screen  in  an  environment 
that  was  dynamically  changing.  There  were  three  main 
sections  in  the  simulation:  the  map  window  on  the  right,  the 
payload  window  on  upper  left,  and  a  status  window  on  the 


Figure  2:  RESCHU  supervisory  control  task.  The  upper  left 
panel  is  the  payload  screen  that  appears  when  a  user  engages  a 
vehicle.  The  bottom  left  panel  provides  the  user  with 
information  about  each  of  the  vehicle  states.  The  right  panel  is 
the  map  view  where  participants  must  navigate  vehicles  to 
targets  while  also  avoiding  hazardous  areas. 


Figure  3:  RESCHU  supervisory  control  task.  The  circles  are 
hazardous  areas.  The  blue  circle  (highlighted  with  a  dotted 
line  container  in  order  to  distinguish  the  cue  when  printing  in 
black  and  white)  demonstrates  what  the  cue  looks  like  when  it 
is  fired.  The  half  circles  represent  what  the  vehicle  looks  like. 
The  red  diamonds  are  the  targets  that  the  vehicles  are  directed 
towards. 


lower  left  (see  Figure  2).  The  map  area  displayed  UAVs 
(blue  half  ovals),  targets  (red  diamonds),  which  UAVs  were 
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directed  to,  and  hazards  (yellow  circles),  which  should  be 
avoided.  Vehicles  were  labeled  with  numbers  and  targets 
were  labeled  with  letters.  The  payload  window  (top  left) 
displayed  a  photographic  image  in  which  the  participant 
engaged  in  visual  search  to  locate  an  object  based  on 
written  instructions  as  part  of  a  payload  delivery  operation 
(described  later).  The  status  window  (bottom  left)  depicted 
a  timeline  of  each  UAV’s  past  and  upcoming  milestones, 
including  the  waypoints  and  the  target  of  each  UAV,  as 
well  as  the  vehicles’  states  (safe,  damaged,  dead).  The 
simulation  included  five  UAVs  that  moved  at  a  fixed  speed, 
5.2  pixels  per  second,  throughout  the  duration  of  the  task. 
There  were  eighteen  hazard  areas,  one  of  which  changed  its 
position  randomly  every  four  seconds,  with  the  constraint 
that  the  hazards  could  not  appear  within  three  degrees  of 
visual  angle  (about  50  pixels)  of  any  UAV.  If  the  UAV 
passed  through  a  hazard,  it  incurred  damage.  Damage  was 
indicated  as  a  red  bar  in  the  status  window.  The  location  of 
targets  and  hazards  on  the  simulation  map  was  randomized 
with  the  constraint  that  targets  and  hazards  were  no  closer 
than  three  degrees  of  visual  angle  from  each  other.  This 
insured  that  targets  and  hazards  could  not  co-occur  in  the 
same  position.  There  were  always  seven  targets  present  on 
the  map. 

Within  the  simulation,  the  system  directed  UAVs  to  targets 
on  straight-line  paths.  The  participant  could  engage  targets 
after  the  vehicle  arrived  at  a  target.  At  the  start  of  the 
simulation  the  UAVs  were  randomly  assigned  to  targets 
towards  which  they  moved  along  automatically  generated 
linear  paths.  Once  the  UAV  reached  the  target  destination, 
the  target  flashed  red  until  it  was  engaged.  A  target  was 
engaged  when  the  operator  right  clicked  on  the  vehicle  and 
selected  the  appropriate  popup  menu  item.  Engaging  the 
vehicle  triggered  the  payload  task,  where  the  participant 
performed  a  visual  search  task  to  locate  an  object  such  as  a 
ship  or  a  car  in  the  payload  window. 

During  the  payload  task,  the  vehicles  in  the  map  panel 
continued  to  move  toward  their  respective  targets,  but 
operator  input  to  the  map  screen  was  disabled.  After 
identifying  the  object  in  the  payload  panel,  the  UAV’s 
mission  was  completed.  The  UAV  was  then  randomly 
assigned  to  a  new  target  that  did  not  already  have  a  UAV 
assigned  to  it. 

The  participant  also  attempted  to  prevent  vehicles  from 
traversing  hazard  areas.  To  avoid  a  hazard  area,  the 
participant  could  assign  the  UAV  to  a  different  target  or  the 
participant  could  add  waypoints  to  the  UAV’s  trajectory, 
which  effectively  allowed  the  participant  to  pilot  the  UAV 
around  hazard  areas.  The  participant  could  also  move  or 
delete  waypoints.  The  trajectory  of  the  UAV  was  indicated 
with  lines  on  the  map,  making  it  unambiguous  whether  a 
UAV  would  traverse  a  hazard  area. 

RESCHU  requires  the  operator  to  manage  multiple  events 
that  occur  in  parallel:  more  than  one  UAV  could  be  waiting 
at  a  target  for  engagement,  multiple  UAVs  could  be  on  a 
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path  to  a  hazardous  area,  and  it  was  left  to  the  operator’s 
discretion  to  act  on  any  one  of  the  five  vehicles.  In  view  of 
these  task  demands,  the  participant  could  not  be  relied  on  to 
notice  whenever  a  UAV  was  on  a  trajectory  towards  a 
hazard. 

In  the  accurate  automation  condition  the  equation  model 
developed  by  Breslow  et  al.  (under  review)  was  run 
repeatedly  (every  500  ms)  in  real-time  with  updated  inputs. 
If  the  model  signaled  danger,  a  highly  salient  cue  flashed  on 
the  hazard  that  the  model  predicted  to  be  posing  a  threat 
(see  Figure  3).  In  the  inaccurate  automation  condition  the 
equation  was  also  run,  but  instead  of  flashing  the  cue  on  the 
relevant  hazard  to  the  model,  a  random  hazard  flashed.  In 
both  conditions  the  cue  was  instantiated  by  the  yellow 
hazard  flashing  blue. 

Design  and  Procedure 

The  experiment  had  a  between  groups  design  with  the 
accurate  automation  and  inaccurate  automation  conditions. 

All  participants  began  the  experiment  by  completing  an 
interactive  tutorial  that  explained  all  aspects  of  the 
simulation.  Participants  learned  about  the  objective  of  the 
simulation:  to  prevent  as  much  damage  as  possible  and 
engage  as  many  vehicles  as  possible.  Additionally, 
participants  learned  how  to  control  the  UAVs  (changing 
targets,  assigning/deleting/moving  waypoints)  and  to 
engage  a  target  (by  right  clicking  on  the  target  and  selecting 
the  engage  menu  item  in  the  popup  menu).  Participants 
were  also  warned  of  the  dangers  of  hazards  and  were 
instructed  on  how  to  avoid  hazards.  The  tutorial  lasted 
approximately  10  minutes. 

After  the  tutorial,  participants  were  instructed  to  practice 
interacting  with  the  RESCHU  simulation  in  the  condition 
that  they  were  assigned  to.  This  practice  was  identical  to 
the  task  that  they  were  later  be  exposed  to.  The 
experimenter  asked  the  participant  to  perform  the  actions 
that  the  participant  was  instructed  on  in  the  tutorial  and  the 
participant  practiced  the  task  until  they  were  comfortable 
with  the  controls.  Participants  were  then  reminded  that  the 
goal  of  the  task  was  to  prevent  as  much  damage  as  possible 
and  engage  as  many  vehicles  as  possible.  Following  this, 
participants  were  calibrated  on  the  eye-tracker,  seated 
approximately  66  cm  from  the  screen,  told  to  try  to  avoid 
damage  as  much  as  possible  and  to  engage  as  many 
vehicles  as  possible,  and  then  were  administered  a  10- 
minute  session  on  RESCHU.  This  session  was  followed  by 
a  brief  break,  after  which  a  second  10-minute  RESCHU 
session  was  administered  in  the  same  manner  as  the  first. 
Participants  were  run  in  the  same  condition  for  both  10- 
minute  sessions. 

Measures 

Keystroke  and  mouse  data  were  collected  for  each 
participant.  Eye  tracking  data  were  collected  using  an  SMI 
eye  tracker  operating  at  250  hertz.  A  fixation  was  defined 
using  the  dispersion-based  method,  where  a  fixation  was 
defined  by  60  ms  of  sample  data  within  a  50  pixel  radius. 
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In  order  to  examine  how  the  aeeuraey  of  the  automation 
impaeted  performanee,  the  pattern  of  eye  movements  was 
analyzed  from  the  moment  the  eue  fired  to  when  it  stopped 
firing,  whieh  either  oeeurred  when  the  partieipant  resolved 
the  danger  situation  or  the  situation  ended  in  damage. 
Fixations  were  eategorized  based  on  their  objeet  of  foeus. 
There  were  a  total  of  five  UAVs  on  the  sereen,  eaeh  having 
a  different  target,  and  possibly  different  hazards  assoeiated 
with  it.  A  vehiele,  the  vehiele’s  relevant  hazard(s),  and  the 
vehiele’s  target  were  elassified  as  a  ‘vehiele  eluster’;  a 
fixation  on  any  of  these  objeets  was  elassified  as  a  fixation 
on  the  vehiele  eluster,  while  the  initial  (and  possibly  only) 
fixation  on  the  hazard  were  elassified  as  a  hazard  fixation. 

To  examine  the  type  of  errors  that  partieipants  made  in  the 
aeeurate  vs.  inaeeurate  automation  eonditions,  three  types 
of  errors  were  distinguished  with  respeet  to  how 
partieipants  responded  to  the  eue:  (a)  Partieipant  did  not 
look  at  the  situation  at  all  (Notieing  Errors),  (b)  Partieipants 
did  not  look  at  the  situation  in  time  to  aet  (Time-out  Errors), 
(e)  Partieipants  looked  at  the  situation  in  time,  deeided  to 
aet  on  something  else,  and  did  not  aet  on  the  eue  situation 
in  time  (Delay  Errors). 

RESULTS 

Performance  on  the  RESCHU  Simulation 

A  mixed  ANOVA  was  run  with  eondition  as  a  between 
groups  faetor  and  session  as  a  within  groups  faetor  in  order 
to  determine  the  impaet  of  the  aeeuraey  of  the  automation 
on  performanee,  where  performanee  was  evaluated  based 
on  the  number  of  instanees  where  a  vehiele  reeeived 
damage  by  making  eontaet  with  a  hazardous  area  on  the 
map.  As  expeeted,  there  were  fewer  instanees  of  damage  in 
the  aeeurate  automation  eondition  (M=  1.86,  CI=  .60)  than 
the  inaeeurate  automation  eondition  (M=  3.95,  Cl  =  1.16), 
F(l,  20)  =  6.89,  p  <  .05,  =  .27.  There  was  not  a 

signifieant  effeet  of  session,  F(l,  20)  =  1.12,  p  =  .30,  = 

.05,  and  no  interaetion  between  eondition  and  session,  F(l, 
20)=  1.53,;?  =  .23,//^  =  .07. 

The  Cue’s  Effect  on  Perceptions  and  Actions 

Reeall  that  we  hypothesized  that  partieipants  would  be 
more  responsive  to  the  eue  in  the  aeeurate  automation 
eondition  than  the  inaeeurate  automation  condition.  In 
support  of  this  hypothesis,  participants  in  the  accurate 
automation  condition  were  more  likely  to  look  at  the  cue  (M 
=  60.55%,  Cl  =  9.56%)  than  participants  in  the  inaccurate 
automation  condition  (M  =  25.21%,  Cl  =  8.44%),  t(20)  = 
5.43, p  <  .05,  d  =  23\  (see  Figure  4,  first  two  bars).  It  also 
might  be  expected  that  participants  would  take  longer  to 
respond  to  the  inaccurate  automation  than  the  accurate  cue, 
yet  there  was  no  difference  in  how  long  it  took  for 
participants  to  respond  to  the  accurate  automation  condition 
(M=  3.79  secs.  Cl  =  .92  secs)  and  inaccurate  automation 
condition  (M=  4.78  secs,  C/=  1.34  secs),  t(2C)  =  \.20,p  = 
.24,  d  =  .51.  There  was  a  marginal  difference  in  the 
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percentage  of  time  participants  fixated  on  the  vehicle 
cluster,  where  participants  in  the  accurate  automation 
condition  were  marginally  more  likely  to  look  at  the 
relevant  vehicle  cluster  after  the  automation  fired  (M  = 
88.83%,  Cl  =  4.07%)  than  participants  in  the  inaccurate 
automation  condition  (M  =  82.73%,  Cl  =  4.86%),  t(20)  = 
1.89,  p  =  .07,  d  =  (see  Figure  4,  middle  two  bars).  In 
further  support  of  the  hypothesis  that  participants  are  more 
responsive  to  the  automation  in  the  accurate  condition  than 
the  inaccurate  condition,  after  looking  at  the  situation 
(vehicle  cluster)  participants  were  more  likely  to  act  and 
resolve  the  hazardous  situation  in  the  accurate  condition  (M 
=  96.75%,  C/=  1.95%)  than  the  inaccurate  condition  (M  = 
86.57%,  Cl  =  8.63%),  t(20)  =  2.26,  p  <  .05,  d  =  .96  (see 
Figure  4,  last  two  bars). 

These  results  confirm  previous  findings  that  showed 
advantages  of  accurate  automation.  Additionally,  the  results 
provide  a  reason  for  why  participants  performed  better  in 
the  accurate  condition  than  the  inaccurate  condition.  In  the 
accurate  condition,  participants  were  more  likely  to  look  at 
the  cue  and  were  more  likely  to  act  on  the  relevant  situation 
than  in  the  inaccurate  condition.  This  demonstrates  that 
accurate  automation  results  in  increased  responsiveness  for 
both  perceptions  and  actions. 


Cue  Effect  in  Accurate  DMOO  and  Inaccurate  DMOO 


A.  Cue  fire  /  look  at  cue  B.  Cue  fire  /  Look  at  sit  C.  Cue  fire  /  Look  at  sit  /  Act 


Figure  4:  Perception  and  action  behavior  based  on  task 
condition.  (A)  The  first  two  bars,  which  represent  the 
percentage  of  time  participants  look  at  the  cue  after  it  fires. 
(B)  The  second  two  bars,  which  represent  the  percentage  of 
time  participants  look  at  the  situation  after  the  cue  fires.  (C) 
The  third  two  bars,  which  represent  the  percentage  of  time 
participants  look  at  the  situation  after  the  cue  fires  and  acted 
on  the  situation.  Error  bars  are  95%  confidence  intervals. 

Types  of  Errors 

While  there  were  fewer  errors  in  the  aeeurate  DMOO 
eondition  than  the  inaeeurate  DMOO  eondition,  as  was 
demonstrated  by  the  previous  finding  of  improved 
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performance  in  the  accurate  DM00  condition,  we 
hypothesized  that  the  distribution  of  errors  would  be  no 
different  between  the  conditions.  The  reason  for  this  is  that 
in  both  conditions  the  cue  was  initiated  using  the  DM00, 
which  fires  the  cue  at  the  same  point  in  time.  The  DM00 
fires  a  cue  at  a  critical  moment,  at  a  later  point  in  time. 
Despite  the  accuracy  of  the  cue,  when  it  is  fired  at  a  later 
point  in  time  this  should  not  impact  the  proportion  of 
noticing  errors,  delay  errors,  and  time-out  errors. 

In  order  to  explore  this,  instances  where  participants  had  no 
errors  were  eliminated  from  the  analysis  and  a  2  X  3  mixed 
ANOVA  was  run  with  error  type  as  a  within  subjects 
independent  variable,  and  with  the  percentage  of  errors  as  a 
dependent  variable.  Again,  condition  was  the  between 
group  variable.  The  types  of  errors  were  noticing  errors, 
time-out  errors,  and  delay  errors.  There  was  not  a 
significant  difference  in  the  percentage  of  errors  between 
conditions,  F(l,  16)  =  1.66,  p  =  .22,  =  .09.  There  was 

also  not  a  significant  in  the  type  of  errors  within  conditions, 
F(2,  32)  =  1.65,  =  .21,  //^  =  .09  and  there  was  no 

interaction  between  condition  and  the  type  of  errors,  F(2, 
32)  =  0.02,  p  =  .98,  =  .00  (see  Figure  5).  While  there 

are  other  explanations  for  this  non-significant  effect,  in 
Experiment  2  we  will  investigate  whether  the  timing  of  the 
cue  impacts  the  proportion  of  error  types. 

DISCUSSION 

As  expected,  participants  were  more  responsive  to  the  cue 
in  the  accurate  automation  condition  than  the  inaccurate 
automation  condition.  Participants  looked  at  the  accurate 
automation  cue  more  often  than  the  inaccurate  automation 
cue  and  were  more  likely  to  act  on  the  accurate  automation 
cue  than  the  inaccurate  automation  cue.  Moreover, 
participants  performed  better  in  the  accurate  automation 
condition  than  the  inaccurate  automation  condition. 

In  support  of  the  idea  that  the  timing  of  automation  impacts 
the  distribution  of  error  types,  there  was  no  difference  in  the 
proportion  of  different  types  of  errors  that  operators  make 
in  the  accurate  cue  and  inaccurate  cue  conditions.  While 
more  errors  occurred  overall  in  the  inaccurate  automation 
condition,  there  was  not  a  significant  difference  in  the 
proportion  of  error  types  in  the  accurate  DM00  and 
inaccurate  DM00.  We  believe  that  there  was  not  a 
significant  difference  in  the  proportion  of  error  types  in 
both  conditions  because  the  cue  fired  at  the  same  time  in 
both  conditions,  resulting  in  the  same  opportunity  to 
respond  to  the  automation  and  then  decide  to  do  something 
else.  Thus,  while  the  accuracy  of  the  automation  affects  the 
number  of  errors  and  performance  because  accurate 
automation  brings  the  participant’s  attention  to  the  relevant 
situation,  it  did  not  significantly  affect  the  distribution  of 
error  types.  In  Experiment  2  we  will  explicitly  test  the 
hypothesis  that  the  timing  of  cue  automation  impacts  the 
distribution  of  error  types  by  manipulating  the  timing  of  cue 
invocation. 
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EXPERIMENT  2 

To  determine  how  automation  invocation  impacts  operator 
error,  and  specifically  how  the  timing  of  AA  impacts  the 
distribution  of  error  types,  we  again  used  the  RESCHU 
simulation  and  the  same  adaptive  model-based  cue 
(Breslow  et  al.  under  review).  Yet  in  this  experiment,  the 
DM00  was  compared  to  a  simpler  system  that  initiated 
automation  based  solely  on  critical  events  invocation  of 
automation.  In  the  critical  event  invocation  of  automation 
condition,  the  hazard  flashed  the  instant  that  a  vehicle 
entered  a  path  intersecting  with  it.  This  was  in  contrast,  to 


Errors  in  Accurate  DM00  and  Inaccurate  DMOO 


Type  of  Error 


Figure  5:  Type  of  error  based  on  task  condition.  Error  bars 
are  95%  confidence  intervals.  A  noticing  error  occurs  when 
the  participant  never  looks  at  the  critical  vehicle,  a  time-out 
error  occurs  when  the  participant  looks  at  the  vehicle,  but 
cannot  act  in  time,  and  a  delay  error  occurs  when  a  participant 
looks  at  the  critical  vehicle,  decides  to  act  on  something  else, 
and  then  does  not  return  to  the  critical  vehicle. 

the  DMOO  where  the  eue  only  flashed  when  the  system 
deteeted  that  there  was  a  high  probability  that  the  operator 
was  not  going  to  respond  to  the  eritieal  event  in  time.  This 
resulted  in  an  earlier  eue  invoeation  for  the  eritieal  event 
system  eondition  and  a  relatively  later  eue  invoeation  for 
the  DMOO  eondition. 

METHOD 

Participants 

Fifty- two  George  Mason  University  undergraduate  students 
partieipated  for  extra  eredit.  All  partieipation  was 
voluntary.  One  partieipants’  data  was  eliminated  due  to 
experimental  error  that  involved  an  issue  with  the  eye- 
traeking  hardware  not  reeording  data.  Six  partieipants’  data 
were  eliminated  due  to  low  eye  traeking  validity.  In  total, 
45  partieipants’  data  were  analyzed.  All  partieipants  had 
normal  or  eorreeted-to-normal  vision. 
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Materials 

Materials  were  identieal  to  the  RESCHU  task  deseribed  in 
Experiment  1,  with  the  exeeption  that  the  inaeeurate 
DM00  eondition  was  replaeed  with  the  eritieal  event 
invoeation  eondition,  whieh  entailed  flashing  a  eue 
immediately  when  a  vehiele’s  path  first  interseeted  a 
hazardous  area. 

Design  and  Procedure 

The  design  and  proeedure  was  identieal  to  Experiment  1, 
with  the  exeeption  that  partieipants  were  randomly  assigned 
to  the  eritieal  event  invoeation  eondition  instead  of  the 
inaeeurate  eondition. 

Measures 

Measures  were  identieal  to  Experiment  1 . 

RESULTS 

Performance  on  the  RESCHU  Simulation 

A  mixed  ANOVA  was  run  with  eondition  as  a  between 
groups  faetor  and  session  as  a  within  groups  faetor  in  order 
to  determine  the  impaet  of  the  type  of  invoeation  on 
performanee,  where  performanee  was  evaluated  based  on 
the  number  of  instanees  where  a  vehiele  reeeived  damage 
by  interseeting  with  a  hazardous  area  on  the  map.  There 
was  no  differenee  in  the  frequeney  of  damage  between  the 
eritieal  event  invoeation  eondition  (M  =  2.70,  Cl  =  1.11) 
and  the  DM00  eondition  (M=  3.77,  CI=  1.77),  F(l,  43)  = 
1.04,  p  =  31,  =  .02.  There  was  a  signifieant  effeet  of 

session,  F(l,  43)  =  53\,  p  <  .05,  =  .11,  and  a  marginal 

interaetion  between  eondition  and  session,  F(l,  43)  =  3.23, 
p  =  .08,  =  .07. 

The  Cue’s  Effect  on  Perceptions  and  Actions 

We  hypothesized  that  partieipants  would  be  equally 
responsive  to  the  eue  in  the  DM00  eondition  and  eritieal 
event  invoeation  eondition.  In  line  with  this  hypothesis, 
there  was  no  differenee  in  the  pereentage  of  time 
partieipants  fixated  on  the  eue  in  the  DM00  eondition  and 
the  eritieal  event  invoeation  eondition,  t(A?>)  =  0.54,  p  <  .59 
(see  Figure  6,  first  two  bars).  However,  partieipants  fixated 
on  the  situation  a  greater  pereentage  of  time  in  the  eritieal 
event  invoeation  eondition  (M=  90.21%,  Cl  =  2.40%)  than 
the  DM00  eondition  (M  =  85.37%,  Cl  =  2.05%),  /(43)  = 
3.00,  p  <  .05,  d  =  1^  (see  Figure  6,  middle  two  bars).  This 
is  likely  due  to  the  eue  simply  appearing  for  a  longer 
amount  of  time  in  the  eritieal  event  invoeation  eondition 
than  the  DM00  eondition,  resulting  in  a  greater  likelihood 
of  fixating  on  the  situation.  In  the  DM00  eondition  the  eue 
fired  on  average  26.84  see  before  the  vehiele  eould 
potentially  interseet  with  the  hazard  and  in  the  eritieal  event 
invoeation  eondition  the  eue  fired  on  average  40.92  see 
before  the  vehiele  eould  potentially  interseet  with  a  hazard 
(p<.05). 

Also  in  line  with  the  hypothesis  that  partieipants  are  equally 
responsive  to  the  eue  in  the  two  eonditions,  after  looking  at 
the  situation,  partieipants  were  equally  likely  to  resolve  the 
situation  in  the  DM00  eondition  (M  =  93.91%,  Cl  = 
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2.61%)  and  the  eritieal  event  invoeation  eondition  (M  = 
93.72%,  Cl  =  2.88%),  t(43)  =  0.25,  p  =  .81,  d  =  .03  (see 
Figure  6,  last  two  bars).  These  results  suggest  that 
partieipants  had  similar  pereeptions  and  aetions  in  response 
to  equally  aeeurate  eues. 

We  also  hypothesized  that  partieipants  in  the  eritieal  event 
invoeation  eondition  would  be  more  likely  to  look  at  the 
eue  and  then  deeide  to  do  something  else,  sinee  they  had 
more  time  available  to  resolve  the  problem.  As  a  result, 
partieipants  in  the  eritieal  event  invoeation  eondition  should 
respond  to  the  eue  more  slowly  than  partieipants  in  the 
DM00  eondition.  In  support  of  this  hypothesis,  while  there 
was  no  differenee  in  how  long  it  took  for  partieipants  to 
look  at  the  eue  in  the  eritieal  event  invoeation  eondition  and 
DM00  eondition,  t(A3)  =  0.34,  p  =  .74,  after  looking  at  the 
eue,  partieipants  took  longer  to  aet  on  the  eritieal  event 
invoeation  eondition  (M=  7.31  sees,  CI=  .99  sees)  than  the 
DM00  eondition  (M  =  5.50  sees.  Cl  =  .59  sees),  t(A3)  = 
3A\,p  <  .05,  (7  =  90  (see  Figure  7).  This  was  likely  due  to 
the  faet  that  partieipants  had  more  time  to  deal  with  the 
event  in  the  eritieal  event  invoeation  eondition  than  the 
DM00  eondition. 

Cue  Effect  in  DMOO  Cue  and  Critical  Event  Invocation  Cue 
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Figure  6:  Perception  and  action  behavior  based  on  task 
condition.  (A)  The  first  two  bars,  which  represent  the 
percentage  of  time  participants  look  at  the  cue  after  it  fires. 
(B)  The  second  two  bars,  which  represent  the  percentage  of 
time  participants  look  at  the  situation  after  the  cue  fires.  (C) 
The  third  two  bars,  which  represent  the  percentage  of  time 
participants  look  at  the  situation  after  the  cue  fires  and  act  on 
the  situation.  Error  bars  are  95%  confidence  intervals. 

The  finding  that  participants  take  longer  to  act  in  the  critical 
event  invocation  condition  (see  Figure  7)  and  are  more 
likely  to  look  at  the  situation  in  the  critical  event  invocation 
condition  (see  Figure  6,  middle  two  bars)  suggests  that 
participants’  strategies  differ  in  response  to  the  two  types  of 
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cues.  While  for  the  DM00  condition  participants  look  at 
the  cue  and  respond  by  quickly  acting,  for  the  eritieal  event 
invoeation  eondition,  partieipants  take  longer  to  respond. 
This  is  due  to  the  faet  that  in  the  eritieal  event  invoeation 
eondition  there  was  a  greater  number  of  instanees  where 
partieipants  looked  at  the  eued  situation  and  deeide  to 
resolve  other  problems  (M  =  12.17,  Cl  =  2.04)  than  in  the 
DM00  condition  (M=  5.32,  Cl  =  1.56),  =  521,  p< 

.05,  (i  =  1 .5 1 .  The  reason  that  partieipant  are  more  likely  to 
delay  dealing  with  the  eued  situation  in  the  eritieal  event 
invoeation  condition  is  likely  beeause  they  have  more  time 
to  return  to  the  eued  problem  later. 


Cue  Effect  in  DMOO  Cue  and  Critical  Event  Invocation  Cue 


How  long  it  takes  to  look  at  cue  How  long  it  takes  to  act  after  looking 


Figure  7:  Timing  of  perception  and  action  behavior  based  on 
task  condition.  Error  bars  are  95%  confidence  intervals. 

Types  of  Errors 

It  was  hypothesized  that  different  proportions  of  error  types 
would  oceur  between  the  DMOO  eue  condition  and  the 
eritieal  event  invoeation  eue  condition  due  to  the  effeet  of 
the  timing  of  the  eue.  An  early  eue,  sueh  as  the  eritieal 
event  invoeation  eue,  increases  the  risk  of  delay  errors, 
while  later,  urgent  cues,  sueh  as  the  DMOO  eue,  inereases 
the  prospeet  of  noticing  errors  and  time-out  errors.  In  order 
to  test  this,  instanees  where  partieipants  had  no  errors  were 
eliminated  from  the  analysis  and  a  2  X  3  mixed  ANOVA 
was  run  with  the  pereentage  of  errors  as  a  dependent 
variable.  Condition  was  between  group  and  type  of  error 
was  within  groups,  with  the  type  of  error  ineluding  noticing 
errors  (not  looking  at  the  eue  at  all),  time-out  errors 
(looking  at  the  eue  but  not  aeting  in  time),  and  delay  errors 
(looked  at  something  else  and  not  returning  in  time).  There 
was  no  effeet  of  eondition,  F(l,  33)  =  0.16,  =  .69,  = 

.00,  and  there  was  a  main  effeet  of  type  of  error,  F{2,  66)  = 
20.02,  p  <  .05,  =  .38,.  In  support  of  the  hypothesis  that 

the  timing  of  the  cue  results  in  different  types  of  errors, 
there  was  an  interaction  between  eondition  and  the  type  of 
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error,  F(2,  66)  =  4.69,  p  <  .05,  =  .14,  where  Benjamini 

Hoehberg  tests  showed  that  delay  errors  were  more 
prevalent  in  the  eritieal  event  invoeation  eondition  than  the 
DMOO  eondition  (p  <  .05),  and  notieing  errors  were 
marginally  more  prevalent  in  the  DMOO  condition  {p  = 
.07).  There  was  no  differenee  between  the  conditions  in  the 
frequency  of  time-out  errors  (see  Figure  8).  Thus,  the  types 
of  errors  differ  between  eonditions  with  notieing  errors 
being  more  eommon  in  the  DMOO  eondition  and  delay 
errors  being  more  eommon  in  the  eritieal  event  invocation 
condition.  This  is  due  to  pereeptual  errors  being  more 
eommon  for  the  DMOO  invoeation  condition  and 
prospective  memory  /  planning  errors  being  more  eommon 
in  the  eritieal  event  invoeation  eondition. 


Error  in  DMOO  Cue  and  Critical  Event  Invocation 


Type  of  Error 

Figure  8:  Type  of  error  based  on  task  condition.  Error  bars 
are  95%  confidence  intervals. 

DISCUSSION 

While  performanee  between  the  DMOO  eondition  and  the 
eritieal  event  invocation  condition  was  the  same, 
participants  responded  to  the  eues  differently  and 
experienced  different  proportions  of  error  types.  For  the 
DMOO  eondition,  partieipants  acted  on  the  vehicle  more 
quickly  after  they  looked  at  the  cue.  This  suggested  that 
participants  responded  to  the  initializations  of  the  eue 
differently,  where  the  DMOO  eondition  resulted  in 
resolving  the  problem  more  quickly  after  it  was  looked  at. 

In  support  of  our  hypothesis  regarding  different  types  of 
errors  based  on  the  timing  of  automation  invoeation,  in  the 
DMOO  eondition,  partieipants  were  more  likely  to  make 
notieing  errors,  i.e.  not  looking  at  the  eue.  Yet  for  the 
eritieal  event  invocation  condition,  participants  were  more 
likely  to  make  delay  errors  due  to  looking  at  the  eue, 
deciding  to  do  something  else,  and  then  not  looking  baek  at 
the  eue.  This  may  be  due  to  forgetting,  or  not  having 
enough  time  to  act  after  returning  to  the  problem  (e.g.  due 
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to  poor  planning).  This  suggested  that  a  later  eue,  whieh 
funetions  as  a  more  urgent  eue,  has  the  disadvantage  of 
produeing  notieing,  pereeptual-based  errors,  while  the 
earlier  eritieal  event  invoeation  has  the  disadvantage  of 
produeing  delay  errors  due  to  degraded  prospeetive 
memory  and  poor  planning. 

CONCLUSION 

We  presented  two  studies  that  eompared  the  DM00  with 
an  inaeeurate  DM00  automation  and  a  simpler  model  that 
invoked  AA  using  only  a  eritieal  event.  The  DM00  is  a 
form  of  AA  that  invokes  automation  using  a  hybrid  method 
that  takes  into  aeeount  both  eritieal  events  in  the 
environment  and  the  operator’s  state  [14].  Of  partieular 
interest  was  how  partieipants  responded  to  the  eomputer 
automation,  given  that  automation  ean  interaet  with  human 
behavior  in  unantieipated  ways  that  ean  have  negative 
eonsequenees  [1,  2,  3,  4]. 

In  support  of  previous  researeh  related  to  the  benefits  of 
more  aeeurate  automation  [5,  19],  partieipants  performed 
better  on  a  eomplex  supervisory  eontrol  simulation  when 
the  DM00  initiated  an  aeeurate  automation  than  when  the 
DM00  initiated  an  inaeeurate  automation.  Partieipants’ 
behavior  suggested  greater  trust  in  the  aeeurate  automation, 
whieh  was  refleeted  by  an  inereased  likelihood  of  fixating 
on  the  aeeurate  eue  and  aeting  on  the  aeeurate  eue. 

In  line  with  of  our  hypothesis  regarding  how  the  timing  of 
automation  invoeation  affeets  the  distribution  of  error  types, 
in  Experiment  1,  when  the  automation  in  both  conditions 
was  invoked  using  the  DM00  that  fires  a  cue  at  an 
identical  time,  there  was  not  a  significant  difference  in  the 
distribution  of  error  types.  Thus,  when  automation  was 
invoked  using  the  DM00,  participants  proportionally  made 
the  same  types  of  errors,  where  these  types  of  errors 
included:  not  looking  at  the  situation  in  time  (noticing 
errors),  looking  at  the  situation  but  not  acting  in  time  (time¬ 
out  errors),  and  looking  at  the  cue,  deciding  to  act  on 
something  else,  and  then  not  returning  to  the  cue  in  time 
(delay  errors). 

Experiment  2  provided  further  support  for  the  hypothesis 
that  the  timing  of  automation  affects  the  distribution  of 
error  types  because  different  proportions  of  error  types 
occurred  between  the  DM00  invocation  condition  and  the 
critical  event  automation  invocation  condition.  When  the 
automation  was  invoked  earlier,  which  occurred  in  the 
critical  events  invocation  condition,  participants  took  longer 
to  act  after  looking  at  the  cue.  This  suggested  that  a  critical 
event  invocation  method  would  result  in  a  greater 
likelihood  of  delay  errors.  Indeed,  this  strategy  of  delaying 
a  response  in  the  critical  event  invocation  condition  resulted 
in  a  greater  proportion  of  errors  due  to  the  participant 
looking  at  the  situation,  deciding  to  act  on  something  else, 
and  not  coming  back  to  the  situation  in  time.  Yet  in  the 
DM00  condition,  errors  were  more  likely  to  occur  due  to 
not  looking  at  the  event  in  time  (noticing  errors). 
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The  different  proportions  of  error  types  in  the  critical  event 
invocation  condition  and  DM00  condition  can  be 
understood  in  terms  of  failures  in  different  levels  of 
information  processing.  When  a  cue  is  initiated  early,  there 
is  a  greater  likelihood  of  errors  due  to  degraded  prospective 
memory,  that  is,  the  inability  to  remember  actions  that  are 
planned  for  the  future  [20],  or  due  to  poor  planning.  Yet 
when  automation  is  invoked  using  the  DM00  method, 
which  only  alerts  participants  when  the  situation  is  critical, 
at  a  later  point  in  time,  participants  were  more  likely  to 
make  perceptual  errors,  where  they  did  not  look  at  the  cue 
before  an  error  occurred. 

The  finding  that  the  invocation  of  AA  impacts  the  types  of 
errors  that  operators  make  has  implications  for  how  to 
improve  operator  performance  in  various  types  of  complex 
and  dynamic  computer  tasks.  Since  prospective  memory 
degrades  when  attention  is  divided  [22,  23],  highly  complex 
tasks  that  tax  working  memory  may  benefit  more  from 
invocation  of  automation  that  occurs  only  at  critical  times, 
such  as  is  the  case  with  the  DM00.  But  tasks  that  are 
simpler  and  require  that  the  participant  be  perceptually 
aware  of  the  environment  may  benefit  more  from  earlier 
cues,  as  in  the  critical  event  invocation  condition  in 
Experiment  2.  These  findings  also  suggest  that  there  may 
be  advantages  to  invoking  multiple  cues.  A  cue  that  occurs 
earlier  could  address  perceptual  errors  and  a  cue  that  occurs 
at  a  critical  moment  could  address  prospective  memory  and 
planning  errors. 

One  important  limitation  regarding  the  effect  of  timing  of 
AA  on  the  types  of  errors  that  operators  make  is  that  these 
finding  may  only  apply  to  dynamic  tasks  where  multiple 
events  occur  in  parallel.  For  example,  in  a  simpler  task 
where  a  single  event  unfolds  in  a  specific  order,  the 
operator  does  not  have  the  opportunity  to  delay  responding 
to  a  cue  by  addressing  other  components  of  the  task. 

Additionally,  the  finding  regarding  the  different  proportion 
of  error  types  based  on  cue  invocation  may  be  limited  to 
specific  task  parameters  and  levels  of  expertise.  For 
example,  perceived  consequences  can  affect  the  operator’s 
willingness  to  make  delay  errors  because  a  more  severe 
consequence  for  making  an  error  may  result  in  the  operator 
being  less  likely  to  delay  addressing  the  problem.  Delay 
errors  may  also  be  less  common  with  more  heavily  trained 
and  experienced  personnel  because  they  may  better 
understand  the  timing  constraints  of  the  task  and  therefore 
be  less  likely  to  make  delay  errors.  Nonetheless,  the 
finding  regarding  the  different  distributions  of  errors  based 
on  the  type  of  cue  invocation  method  speaks  to  various 
dynamic  and  time  sensitive  tasks  where  multiple  events 
must  be  managed  in  parallel,  such  as  air  traffic  control, 
piloting  an  airplane,  driving  an  automobile,  and  operating  a 
power  plant.  System  designers  must  consider  how  the 
timing  of  automation  invocation  has  differential  effects  on 
types  of  errors  by  invoking  automation  earlier  to  resolve 
perception  errors  and  later  to  resolve  delay  errors. 
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